feat(calibration): backfill the corpus from historical review_targets decisions by JSONbored · Pull Request #8169 · JSONbored/loopover

JSONbored · 2026-07-23T08:02:42Z

Summary

Ships calibration: backfill the corpus from the 10k+ historical review ledger (labels + firings; raw context phase 2) #8157 phase 1 — and the backfill is already applied to production (per the issue's dry-run-report-then-apply protocol, both recorded on the thread): the pure core synthesizes the signal.rule_fired/signal.human_override pairs the calibration: wire linked_issue_scope_mismatch into the shared signal-tracking module (fired + reversal) #8101 capture writers would have produced from historical review_targets decisions — close verdicts with decision-level confidence, labeled by terminal outcome, mapped per the ratified option (a) onto ai_consensus_defect (one rule id, never duplicated), every row tagged backfilled: true + provenance.
Integrity properties, each pinned by tests: never fabricates (non-close verdicts, missing confidence, non-terminal rows are counted and skipped; modelResponseText never synthesized), idempotent by construction (deterministic ids + chunked INSERT OR IGNORE; re-runs and partial-apply overlaps are no-ops), latest-terminal-decision-wins dedupe, and both SQLite/ISO timestamp eras normalized. A buildBacktestCorpus round-trip test proves synthesized rows become real labeled BacktestCases.
Production result: 460 eligible decisions → 920 rows applied; verified live by backtest-corpus-export round-tripping 460 labeled cases (checksum bd92485cea22…). The threshold backtest and the calibration trend now start from real history instead of an empty corpus.

Advances #8157 (phase 2 — GitHub-truth reversal labels + raw-context re-fetch — is being split to its own issue; #8157 closes once this merges).

Scope

The PR title follows type(scope): short summary Conventional Commit format, for example fix(api): restore profile access checks.
This PR is focused and does not mix unrelated backend, UI, MCP, docs, dependency, and deploy changes.
This follows CONTRIBUTING.md and does not reintroduce GitHub Pages, VitePress, site/, or CNAME.
I linked a currently open issue this PR resolves (e.g. Closes #123) — a linked open issue is required for every contributor PR.

Validation

If any required check was skipped, explain why:

Full npm run test:ci ran green through the whole suite (1108 test files, 0 failures); its ui:version-audit failure was purely the branch's stale fork point, resolved by rebasing onto current main (audit re-ran green, plus typecheck + the new suite). The diff is three standalone scripts/** + test/unit/** files — outside coverage.include per the established scripts exemption, with 100% line AND branch coverage on the pure core anyway (10 tests).

Safety

No secrets, wallet details, hotkeys, coldkeys, user PATs, private keys, raw trust scores, private rankings, or private maintainer evidence are exposed.
Public GitHub text stays sanitized, low-noise, and does not imply compensation guarantees or optimization tactics.
Auth, cookie, CORS, GitHub App, Cloudflare, or session changes include negative-path tests.
API/OpenAPI/MCP behavior is updated and tested where needed.
UI changes use live API data or real empty/error/loading states, not production mock/demo fallbacks.
Visible UI changes include a UI Evidence section below with JPG/JPEG or PNG screenshots arranged as organized, captioned, clickable thumbnails. SVG screenshots are not used as review evidence. Review-only screenshots or recordings are not committed to the repository.
Public docs/changelogs are updated where needed; changelogs are only edited for release-prep PRs.

Backfilled rows carry only what the ledger already held (verdict, confidence, timestamps) plus tags — no raw content of any kind is synthesized or copied.

UI Evidence

Not applicable — no UI change.

Notes

Reversible by tag: DELETE FROM audit_events WHERE json_extract(metadata_json,'$.backfilled') removes the era cleanly if ever needed.

superagent-security · 2026-07-23T08:03:41Z

Superagent didn't find any vulnerabilities or security issues in this PR.

codecov · 2026-07-23T08:12:00Z

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 92.06%. Comparing base (87e8a89) to head (fc2b53b).
⚠️ Report is 3 commits behind head on main.
✅ All tests successful. No failed tests found.

Additional details and impacted files

@@           Coverage Diff           @@
##             main    #8169   +/-   ##
=======================================
  Coverage   92.06%   92.06%           
=======================================
  Files         768      768           
  Lines       77838    77838           
  Branches    23516    23517    +1     
=======================================
  Hits        71663    71663           
  Misses       5062     5062           
  Partials     1113     1113

Flag	Coverage Δ
shard-1	`56.99% <ø> (-0.23%)`	⬇️
shard-2	`53.52% <ø> (+0.98%)`	⬆️
shard-3	`51.99% <ø> (-1.73%)`	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

loopover-orb · 2026-07-23T08:20:28Z

Caution

🛑 LoopOver review result - fixes required

_{Review updated: 2026-07-23 08:24:30 UTC}

3 files · 1 AI reviewer · 2 blockers · CI failing · blocked

🛑 Suggested Action - Manual Review

Review summary
Adds a pure-core + thin-IO CLI pair that reconstructs signal.rule_fired/signal.human_override audit rows from historical review_targets close decisions, mapping them onto a single ai_consensus_defect rule id per the ratified option (a), with a 150-line test suite exercising every eligibility branch (wrong verdict, missing/NaN confidence, non-terminal status, duplicate-target dedupe, both timestamp eras, and a round-trip through buildBacktestCorpus). The core logic is correct: eligibility filtering, descending sort by normalized terminal timestamp, and latest-wins dedupe by targetKey all match the stated integrity rules, and the CLI is a straightforward dry-run-by-default wrapper mirroring the existing backtest-corpus-export.ts split. The most notable subtlety is that idempotency across separate invocations relies on a targetKey-only deterministic id combined with INSERT OR IGNORE, so if a target's terminal decision changes between two backfill runs, the second run's INSERT is silently dropped rather than updating the existing row — acceptable for a one-time historical backfill of already-terminal PRs but worth calling out explicitly.

Nits — 5 non-blocking

scripts/backfill-calibration-corpus-core.ts: the synthesized row id (`backfill:${rule}:${targetKey}:fired/override`) is keyed only on targetKey, not on the decision content, so if review_targets' terminal decision for a given PR changes between two backfill invocations, the second run's INSERT OR IGNORE silently keeps the stale first-run row instead of updating it — worth a one-line comment noting this is safe only because the source data is already terminal and won't change.
scripts/backfill-calibration-corpus-core.ts:160 and scripts/backfill-calibration-corpus.ts:39/42 — the chunk size (50), maxBuffer (256*1024*1024), and similar constants are unnamed magic numbers; a named constant would make the intent (wrangler --command size limits) clearer.
scripts/backfill-calibration-corpus.ts:d1Execute — JSON.parse(result.stdout) has no try/catch, so a malformed/truncated wrangler JSON response throws an unhandled SyntaxError instead of the intended fail-loud Error with stderr context.
The two failing CI checks (`validate`, `validate-tests (1)`) are reported with no detail — I can't verify the cause from what's given here; worth checking the actual failure output before merging.
Consider naming the magic numbers (50-row chunk size, 256MB buffer) as exported constants for discoverability, per scripts/backfill-calibration-corpus-core.ts:160 and scripts/backfill-calibration-corpus.ts:39.

Why this is blocked

No linked issue detected: No closing reference or linked issue number was found in the PR metadata/body. — If this PR is intended to solve an issue, link it explicitly in the PR body.
Maintainer requires a linked issue: This repo's maintainer focus manifest requires every PR to reference a tracked issue. — Link the relevant issue (for example Closes #123) before opening the PR.

📋 Copy for AI agents — paste into your coding agent

Fix the following blocker(s) from this PR review:

1. No linked issue detected: No closing reference or linked issue number was found in the PR metadata/body. — If this PR is intended to solve an issue, link it explicitly in the PR body.

2. Maintainer requires a linked issue: This repo's maintainer focus manifest requires every PR to reference a tracked issue. — Link the relevant issue (for example `Closes #123`) before opening the PR.

CI checks failing

validate
validate-tests (1)

Decision drivers

❌ Code review — 2 blockers (1 reviewer)
❌ Gate result — Blocking (Repo-configured hard blocker found.)

Context & advisory signals — never blocks the verdict

Signal	Result	Evidence
Linked issue	⚠️ Missing	No linked issue or no-issue rationale found.
Related work	✅ No active overlap found	No same-issue or scoped active PR overlap found.
Change scope	✅ 20/20	Low review scope from cached public metadata (no linked issue context).
Validation posture	✅ 25/25	PR body includes validation/test evidence.
Contributor workload	✅ 10/10	Author activity: 14 registered-repo PR(s), 14 merged, 253 issue(s).
Contributor context	✅ Confirmed Gittensor contributor	JSONbored; Gittensor profile; 14 PR(s), 253 issue(s).
Improvement	✅ Minor	risk: clean · value: minor

Review context

Author: JSONbored
Role context: owner (maintainer lane)
Public audience mode: oss maintainer
Lane context: Repository is configured for direct PR review.
Public profile languages: Python, TypeScript, Ruby, Go, JavaScript, MDX, Shell, Solidity
Official Gittensor activity: 14 PR(s), 253 issue(s).
PR-specific overlap: none found.

Contributor next steps

Start here: Treat this as maintainer-lane context rather than normal contributor-lane activity.
Then work through the remaining 2 steps in the Signals table above.

Signal definitions

Related work = same linked issue, overlapping active PRs, or title/path similarity.
Change scope = cached public metadata such as size labels, draft state, and review-burden hints.
Validation posture = whether the PR provides enough public validation/test evidence for maintainer review.
Contributor workload = public contributor activity and cleanup pressure, not a repo-wide quality failure.
Contributor context = public GitHub/Gittensor identity context; non-Gittensor status is not a blocker.

🧪 Chat with LoopOver

Ask LoopOver a question about this PR directly in a comment — grounded only in the same cached, public-safe facts shown above, never a new claim.

@loopover ask <question> answers contribution-quality Q&A with source citations and freshness.
@loopover chat <question> answers in natural prose from cached decision-pack facts via local inference (maintainer/collaborator; read-only).
A plain-language @loopover mention with a real question is routed to the closest matching read-only command automatically — no exact syntax required.

Full command reference: https://loopover.ai/docs/loopover-commands

🧪 Experimental — new and may change.

_{🟩 Safe / merged · 🟦 Advisory · 🟨 Held for review · 🟥 Blocked / closed}

💰 Earn for open-source contributions like this. Gittensor lets GitHub contributors earn for the work they already do — register to start earning →.

Checked by LoopOver, a quiet PR intelligence layer for OSS maintainers.

Re-run LoopOver review

JSONbored · 2026-07-23T08:26:27Z

Good catch by the review on the INSERT OR IGNORE idempotency subtlety — addressed with a real fix rather than a comment: buildBackfillInsertStatements now emits ON CONFLICT(id) DO UPDATE on the decision-bearing columns (detail, metadata_json, created_at), so a target whose terminal decision changes between runs gets its pair updated (matching the transform's own latest-terminal-wins dedupe), while identical re-runs remain effective no-ops and the deterministic ids/provenance are unchanged. Test updated to pin the conflict clause. The already-applied production rows are unaffected (a re-run would update them to identical values).

… decisions (#8157 phase 1) The calibration corpus only accumulated from the #8101 capture writers' ship date, while review_targets held 3,233 historical decisions. The pure core synthesizes the fired/override pairs those writers would have produced: close-verdict decisions with decision-level confidence, labeled by terminal outcome (closed = confirmed; a close-verdict PR that ended merged = reversed), mapped per #8157's ratified option (a) onto ai_consensus_defect — one rule id, never duplicated across siblings — with backfilled+provenance tags on every row so consumers can include or exclude the era explicitly. Never fabricates: non-close verdicts, missing confidence, and non-terminal rows are counted and skipped; modelResponseText is never synthesized. Idempotent by construction (deterministic ids + INSERT OR IGNORE, chunked statements). Applied to production after the dry-run report on the issue: 460 eligible decisions -> 920 rows; verified round-tripping through backtest-corpus-export into 460 labeled BacktestCases. 100% line+branch coverage on the core, including a buildBacktestCorpus round-trip test. Advances #8157.

…lently dropping a changed terminal decision ORB-review finding on the original INSERT OR IGNORE shape: with targetKey-only deterministic ids, a target whose terminal decision changed between two backfill runs had its second run's row silently dropped. ON CONFLICT(id) DO UPDATE on the decision-bearing columns makes cross-run semantics match the transform's own latest-terminal-wins dedupe; identical re-runs remain effective no-ops, and ids/provenance are unchanged.

JSONbored self-assigned this Jul 23, 2026

loopover-orb Bot added the gittensor:bug Gittensor-scored bug fix — scores a 0.05x multiplier. label Jul 23, 2026

loopover-orb Bot added the manual-review Gittensor contributor context label Jul 23, 2026

JSONbored force-pushed the feat/calibration-backfill branch from df5ce39 to c5c5f94 Compare July 23, 2026 08:42

JSONbored added 2 commits July 23, 2026 01:46

JSONbored mentioned this pull request Jul 23, 2026

calibration: backfill the corpus from the 10k+ historical review ledger (labels + firings; raw context phase 2) #8157

Closed

3 tasks

JSONbored force-pushed the feat/calibration-backfill branch from c5c5f94 to fc2b53b Compare July 23, 2026 08:52

JSONbored merged commit 1302171 into main Jul 23, 2026
2 checks passed

JSONbored deleted the feat/calibration-backfill branch July 23, 2026 08:52

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

feat(calibration): backfill the corpus from historical review_targets decisions#8169

feat(calibration): backfill the corpus from historical review_targets decisions#8169
JSONbored merged 2 commits into
mainfrom
feat/calibration-backfill

JSONbored commented Jul 23, 2026

Uh oh!

superagent-security Bot commented Jul 23, 2026

Uh oh!

codecov Bot commented Jul 23, 2026 •

edited

Loading

Uh oh!

loopover-orb Bot commented Jul 23, 2026 •

edited

Loading

🛑 LoopOver review result - fixes required

Uh oh!

JSONbored commented Jul 23, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Uh oh!

Conversation

JSONbored commented Jul 23, 2026

Summary

Scope

Validation

Safety

UI Evidence

Notes

Uh oh!

superagent-security Bot commented Jul 23, 2026

Uh oh!

codecov Bot commented Jul 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

loopover-orb Bot commented Jul 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🛑 LoopOver review result - fixes required

Uh oh!

JSONbored commented Jul 23, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

codecov Bot commented Jul 23, 2026 •

edited

Loading

loopover-orb Bot commented Jul 23, 2026 •

edited

Loading